Differential illumination for corneal glint detection

ABSTRACT

An apparatus, system, and method for detecting glints includes a pair of light sources positioned on a head-mounted frame and driven to illuminate a corneal surface of an eye with differential light signals. An image sensor is positioned on the head-mounted frame to receive reflections of the differential light signals from the corneal surface. An orientation of a cornea (e.g., corneal sphere) of the eye may be determined at least partially based on the reflections of the differential light signals.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. provisional Application No.63/234,621 filed Aug. 18, 2021, which is hereby incorporated byreference.

TECHNICAL FIELD

This disclosure relates generally to glint detection, and in particularto illumination for glint detection systems.

BACKGROUND INFORMATION

Eye tracking inaccuracies can undermine a user's trust in a system thatrelies on an eye tracking system. For example, if an eye tracking systemis used for a head-mounted display, inaccurate eye tracking could makethe system less enjoyable to use or less functional.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the invention aredescribed with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified.

FIG. 1 illustrates a glint detection system, in accordance with aspectsof the disclosure.

FIG. 2 illustrates an ocular environment that includes exampleimplementations of a glint detection system, in accordance with aspectsof the disclosure.

FIG. 3 illustrates an example graph of potential metrics that are basedon light source spacing in the disclosed glint detection system, inaccordance with aspects of the disclosure.

FIG. 4 illustrates a flow diagram of a process for acquiring and usingglint locations, in accordance with aspects of the disclosure.

FIG. 5 illustrates a flow diagram of a process for detecting glints, inaccordance with aspects of the disclosure.

FIG. 6 illustrates a head mounted display, in accordance with aspects ofthe disclosure.

DETAILED DESCRIPTION

Embodiments of a glint detection system using differential illuminationare described herein. In the following description, numerous specificdetails are set forth to provide a thorough understanding of theembodiments. One skilled in the relevant art will recognize, however,that the techniques described herein can be practiced without one ormore of the specific details, or with other methods, components,materials, etc. In other instances, well-known structures, materials, oroperations are not shown or described in detail to avoid obscuringcertain aspects.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

In aspects of this disclosure, visible light may be defined as having awavelength range of approximately 380 nm to 700 nm. Non-visible lightmay be defined as light having wavelengths that are outside the visiblelight range, such as ultraviolet light and infrared light. In aspects ofthis disclosure, red light may be defined as having a wavelength rangeof approximately 620 to 750 nm, green light may be defined as having awavelength range of approximately 495 to 570 nm, blue light may bedefined as having a wavelength range of approximately 450 to 495 nm, andinfrared light may be defined as having a wavelength range ofapproximately 700 nm to 1 mm.

In aspects of this disclosure, a glint may be defined as a cornealreflection that is a virtual image of a light source (e.g., infrared)that illuminates the eye and that is created by the outward facingsurface (i.e., corneal surface) of the cornea. The corneal reflection isa specular reflection off of the corneal surface, which acts as a convexmirror.

Embodiments of the present disclosure include a glint detection systemthat uses differential illumination to reduce image saturation caused bythe illumination of diffuse surfaces, such as skin, an iris, or asclera. When a system is configured to use corneal reflections todetermine the orientation of the eye, a better signal-to-noise ratio canresult in more accurate detection of glints and therefore a moreaccurate identification of gaze orientation. Depending upon theapplication, accurate identification of gaze orientation can provide amore reliable, trustworthy, and enjoyable user experience. For example,a glint detection system that is integrated into an eye tracking systemof a head-mounted display may be used to adjust a focal point of adisplay, may be used to customize user interface elements, and/or may beused to interact with or control applications that are executed with thehead-mounted display.

A glint system may include one or more pairs of light sources, an imagesensor, and glint detection logic, according to an embodiment. The glintdetection logic may be communicatively coupled to the pairs of lightsources and to the image sensor. The glint detection logic may drive thepairs of light sources to emit differential light signals with apattern, such as a square wave. The differential light signals may beswitched in opposition polarities and cause reflections on the cornea.Advantageously, by switching in opposite directions, the differentiallight signals cause diffuse surfaces to maintain a net illumination thatis approximately constant. The image sensor (e.g., an event camera) canbe configured to identify changes in an image scene while ignoringportions of the image scene that remain the same. Without usingdifferential illumination, driving a light source with a square wave cancause the image sensor to saturate, so that glints are barely detectableor undetectable. By generating glints with the differential lightsignals, the image sensor detects changes in the image scene as eventsthat represent the corneal glints with strong signal-to-noise ratios.The image sensor transmits the captured events to the glint detectionlogic as image data, to identify locations of the glints in the imagescene or locations of the glints on the corneal surface. The glintlocations may be mapped (e.g., using machine learning) to a cornealorientation and used to identify an orientation of the corneal sphere.

The glint detection system may be used in an eye tracking system that isincorporated into a head-mounted display, according to an embodiment.The eye tracking system may be configured to identify the pupil center(or other pupil characteristics) of an eye using one or more additionalimage sensors and light sources. The eye tracking system may use theglint locations and/or the pupil center to determine (e.g., calculate) agaze vector or a gaze orientation. The gaze orientation may be providedto the head-mounted display to enable the head-mounted display topersonalize a user's experience based on the user's gaze orientation.

The apparatus, system, and method for a glint detection system isdescribed in this disclosure and enables improved determination ofcorneal sphere orientation and eye tracking in, for example, ahead-mounted display. These and other embodiments are described in moredetail in connection with FIGS. 1-6 .

FIG. 1 illustrates an example of a glint detection system 100, accordingto an embodiment of the disclosure. Glint detection system 100 isconfigured to drive a differential illumination pattern onto a pair oflight sources to generate specular reflections of the corneal surface ofan eye (i.e., glints) while concurrently generating an approximatelyconstant net illumination from diffuse portions of a user's eyes andface. The specular reflections combined with the constant diffuseillumination enables glint detection and identification of glintlocations as a user's cornea moves. Glint detection system 100 isconfigured to reduce the amount of data captured and processed fromimage scenes by monitoring the changes to an image scene while ignoringthe portions of the image scene that remain relatively unchanged.Advantageously, the disclosed differential lighting systems andprocesses enhance the detectability of corneal surface specularreflections, which enables the determination of corneal and gazeorientation, according to embodiments of the disclosure.

Glint detection system 100 includes glint detection logic 102, lightsources 104 (individually, light source 104A and 104B), and image sensor108 carried by a head-mounted frame 110, according to an embodiment.Glint detection logic 102 includes circuitry, processor logic, memory,and/or instructions that support operation of glint detection system100. Glint detection logic 102 is communicatively coupled to lightsources 104 and image sensor 108 with communication channels 112. Glintdetection logic 102 may include or use an illumination pattern 114 togenerate image data 106 and determine glint locations 116, according toan embodiment.

Glint detection logic 102 is configured to drive illumination pattern114 onto light sources 104 to cause light sources 104 to emitdifferential light signals 118 (individually, differential light signal118A and 118B) toward a user 120. User 120 includes diffuse surfaces andreflective surfaces. Some of the diffuse surfaces/objects include theskin, the iris, and the sclera of user 120, which are approximatelyLambertian surfaces that scatter light diffusely. An eye 122 of user 120includes a corneal surface 124. Corneal surface 124 is a specular(reflective) surface that produces a mirror-like reflection ofdifferential light signals 118. Differential light signals 118 arediffusely scattered off of the diffuse surfaces and appear as glints inan image by reflecting off of corneal surface 124.

Differential light signals 118 include a high state H and a low state L.High state H represents a level of illumination that is greater than thelevel of illumination of low state L. Low state L may be the level ofillumination of a light source that is turned off, and high state H maybe the level of illumination of a light source that is turned on,according to an embodiment. Low state L may be the level of illuminationof a light source that is turned on and emits, for example, a lightsignal that is less than 20-30 lumens, and high state H may be the levelof illumination of a light source that is turned on and emits, forexample, anything over 50 lumens, according to an embodiment. In otherwords, high state H and low state L may represent two different levelsof illumination that image sensor 108 may be configured to distinguishbetween. Although example values of illumination are described in termsof lumens, digital pixel values or another metric of illuminationdetected by image sensor 108 may be used.

Light sources 104 emit differential light signals 118 according toillumination pattern 114, according to an embodiment. Illuminationpattern 114 may define a frequency, waveform shape, and/or duty cycle ofdifferential light signals 118. For example, differential light signals118 may be emitted with a square wave pattern, with a duty cycle of 50%(high and low for the same duration) that is repeated at a frequency of1 kHz. With this pattern, light source 104A may be operated to initiallyemit differential light signal 118A with high state H, and in acomplementary configuration, light source 104B may be operated toinitially emit differential light signal 118B with a low state L, priorto switching back and forth between high and low states. In someembodiments, differential light signals 118 may have a frequency in therange of 1 kHz to 2 kHz, or a frequency that is greater than or equal to1 kHz to reduce blurring of glint detection. In an embodiment,differential light signal 118A may have a duty cycle that is less than50% while complementary differential light signal 118B has a duty cyclethat is greater than 50%. In an embodiment, the combined duty cycles ofdifferential light signals 118 is 100%.

Light sources 104 are positioned on head-mounted frame 110 and have anintra-light source angle θ, according to an embodiment. Intra-lightsource angle θ may be used to define how far apart light source 104A andlight source 104B are positioned from each other on head-mounted frame110. For example, as disclosed below in connection with FIG. 3 ,intra-light source angle θ may determine a signal-to-noise ratio (SNR)for glint detection and may determine the proximity of clusters ofglints to each other. In an embodiment, light sources 104 are positionedapart from each other to define intra-light source angle θ to be atleast 2.5 degrees, so clusters of glints captured by image sensor 108are adjacent to each other and not overlapping. In some embodiments,light sources 104 are integrated into lenses of a head-mounted device,are positioned on head-mounted frame 110, or both.

Light sources 104 may be implemented with a variety of types of lightsources and in a number of configurations on head-mounted frame 110, inaccordance with embodiments of the disclosure. Light sources 104 may beimplemented as light-emitted diodes (LEDs), vertical external-cavitysurface-emitting lasers (VCSELs), fiber optics, out-coupling gratings,or the like. Light sources 104 may be positioned to be above eye 122,below eye 122, near the nose, away from the nose, within a field of viewof eye 122 (e.g., mounted or integrated into lenses of AR/VR glasses),and/or outside of a field of view of eye 122 (periphery) on head-mountedframe 110. Although a single pair of light sources 104 is illustratedand discussed in FIG. 1 , multiple pairs of light sources may beincorporated into glint detection system 100, as shown in FIG. 2 anddescribed below.

Glint detection logic 102 is configured to determine glint locations 116at least partially based on image data 106 received from image sensor108, according to an embodiment. Image sensor 108 is positioned onhead-mounted frame 110 a distance D from eye 122. Distance D maypartially be defined by hardware that mounts head-mounted frame 110 tothe head of user 120. Image sensor 108 is oriented to receivereflections 126 (individually, reflection 126A and 126B) of differentiallight signals 118 that reflect off of corneal surface 124. Image sensor108 includes a number of pixels 128 that are photosensitive elementsthat convert light into an electrical signal. Front view 130 of imagesensor 108 illustrates pixels 128 arranged in an addressable pixel array132 of rows R and columns C. Image sensor 108 may include millions ofpixels (e.g., 8 megapixels) to provide resolution in glint detection.

Image sensor 108 may be implemented as an event camera, a complementarymetal oxide semiconductor (“CMOS”) image sensor, a charge-coupled device(“CCD”) image sensor, a time-of-flight sensor, or another photosensitivesensor. When implemented as an event camera, image sensor 108 detectschanges in an image scene without capturing portions of the image scenethat are unchanged. More specifically, as an event camera, image sensor108 captures data from specific pixels that are illuminated above aparticular threshold of illumination. Examples of thresholds ofillumination can be in terms of brightness in lumens or brightness indigital pixel values (dp). Examples of these thresholds may depend onthe bit depth of the image sensor (e.g., 8 bit, 10 bit, 20 bit) and maybe 50 lumens, 100 lumens, 150 lumens, 20 dp, 50 dp, 100 dp, etc. Thethreshold value that defines whether a glint is detected may be adjustedbased on environmental noise (e.g., external sunlight), according to anembodiment. When implemented as a CMOS or CCD image sensor, image sensor108 may include on-chip circuitry to support event-based operations thatignore portions of an image scene that remain relatively constant andthat captures or transmits events associated with pixels that areilluminated to a level that exceeds a pre-determined threshold.

Image data 106 is data that represents glints or clusters of glintscaptured by image sensor 108. Image data 106 includes events that mayrepresent one or more of pixels 128 that exceed a threshold. An eventmay include a pixel address, a time stamp, a digital pixel value, and/ora polarity (e.g., high or low). Image data 106 may be transmitted aseach event is detected, may be transmitted periodically (e.g., every 10ms), or may be transmitted each time a certain number of events aredetected (e.g., every 10 events).

Front view 130 of image sensor 108 depicts an example mapping of pixelclusters that may represent glints detected by image sensor 108. Pixelcluster 134 represents a number of events or glints detected fromreflection 126A of differential light signal 118A, and pixel cluster 136represents a number of events or glints detected from reflection 126B ofdifferential light signal 118B. Pixel cluster 134 and pixel cluster 136represent events or glints detected around a first time t1, which mayinclude a small period of time (e.g., 0.1 ms). Depending upon the sizeand/or pitch of pixels 128, detection of a single event (e.g., a changefrom low-to-high of one of differential light signals 118) may causeseveral pixels 128 to register a digital pixel value that exceeds athreshold. Glint detection logic 102 may be configured to trackindividual events, or glint detection logic 102 may be configured tointegrate several events (e.g., 20-50 events) together prior to updatingglint locations 116, according to various embodiments. Hence, pixelcluster 134 and pixel cluster 136 may be representative of a singledetected event or of several integrated events, according to variousembodiments. As illustrated, at a second time t2 (which may be a smallperiod of time), pixel cluster 134 and pixel cluster 136 have, forexample, changed locations within pixel array 132, which may beindicative of a change of orientation of a corneal sphere 138 of whichcorneal surface 124 is a part of.

Glint detection logic 102 may use one or more synchronization pulses toassociate differential light signals 118 with reflections 126, accordingto an embodiment. For example, glint detection logic 102 may transmit ashort synchronization pulse to light sources 104 and to image sensor 108at the beginning of each period of illumination pattern 114 to correlatepatterns emitted with patterns received.

Glint detection system 100 may be incorporated into one or more systemsto support operations of those systems. As described above, glintdetection system 100 may be configured to use differential light signals118 and image sensor 108 (e.g., an event camera) to dynamically identifyglint locations 116, which change as eye 122 changes orientation.Accordingly, glint detection system 100 may be used in an eye trackingsystem to determine a gaze orientation of a user and may be used in ahead-mounted display (HMD) to adjust a display's focal point,brightness, user interface, etc., at least partially based on where auser's eyes are oriented within the HMD.

FIG. 2 illustrates an ocular environment 200, in accordance withembodiments of the disclosure. Ocular environment 200 illustratesfeatures of glint detection system 100 being integrated into an eyetracking system 202 that is further integrated into an HMD 204,according to an embodiment. Ocular environment 200 includes multiplepairs of light sources and multiple image sensors included in HMD 204 tosupport glint detection and eye tracking.

Ocular environment 200 may include one or more pairs of light sources206 (in addition to light sources 104) that are configured to emitdifferential light signals 208, according to an embodiment. Pairs oflight sources 206 are mounted to head-mounted frame 110 and may bemounted in a variety of locations (e.g., periphery, in-field, etc.).Pairs of light sources 206 and light sources 104 emit light in theinfrared (e.g., near infrared) wavelength range, according to anembodiment. Glint detection logic 102 may be configured to drive lightsources 104 and pairs of light sources 206 with illumination pattern114, according to an embodiment. Illumination pattern 114 may set eachof differential light signals 208A, 208B, 208C, 118 to be different foreach pair of light sources 206A, 206B, 206C and light sources 104. Forexample, illumination pattern 114 may define a different periodicfrequency for each of differential light signals 118, 208A, 208B, and208C, according to an embodiment. For example, illumination pattern 114may define that differential light signals 118 are emitted at 1 kHz,differential light signals 208A are emitted at 1.3 kHz, differentiallight signals 208B are emitted at 1.6 kHz, and differential lightsignals 208C are emitted at 1.9 kHz. Glint detection logic 102 and/oreye tracking system 202 can be configured to correlate light sources 104and pairs of light sources 206 with their corresponding return signals(e.g., glints or events) by, for example, applying a Fourier transformto the detected return signals to identify a frequency of the returnsignals, according to an embodiment.

Ocular environment 200 may also include an image sensor 210 (in additionto image sensor 108) that is configured similarly to image sensor 108 tosupport glint detection. In one embodiment, image sensor 210 isconfigured to capture images of a pupil 212. Image sensor 210 may use adedicated light source or light source pair to capture images of pupil212 and may be configured to capture pupil images concurrently withdetected events, concurrently with the detection of a number of events(e.g., 50), and/or periodically. Image sensor 210 and image sensor 108may include bandpass filters that pass infrared light and filter outother wavelengths.

Eye tracking system 202 may use glint locations 116, corneal map data214, and pupil characteristics 216 to determine a gaze orientation 218,according to an embodiment. Corneal map data 214 may include correlationmappings between glint locations 116 and an orientation of cornealsphere 138, according to an embodiment. Pupil characteristics 216 mayinclude a shape, size, or center of pupil 212 that are determined fromimage data from image sensor 210, for example. Eye tracking system 202may apply various techniques to determine a vector of gaze orientation218 that is at least partially based on, for example, pupilcharacteristics 216 (e.g., a pupil center) and glint locations 116,according to an embodiment.

Eye tracking system 202 may provide gaze orientation 218 to HMD 204 tosupport various operations, according to an embodiment. For example, HMD204 may use gaze orientation 218 to customize user interface elementsused by a user interface 220 and/or to define operations of one or moreapplications 222, according to an embodiment. HMD 204 may use gazeorientation 218 to at least partially drive a display 224, according toan embodiment. For example, based on gaze orientation 218, display 224may adjust brightness, a focal point, or features included in displaylight 226 (e.g., user interface elements).

HMD 204 includes a lens assembly 228 that transmits display light 226 toeye 122, according to an embodiment. Lens assembly 228 may be carried byhead-mounted frame 110 within HMD 204 and may include one or morelenses, grates, and/or other optical elements. One or more pairs oflight sources 206 and/or light sources 104 may be integrated into lensesof lens assembly 228 to provide in-field (e.g., within a field of viewof eye 122) illumination.

HMD 204 includes processing logic 230 and memory 232, according to anembodiment. Processing logic 230 and/or memory 232 may includeinstructions 234 that are machine-readable and executable by processinglogic 230 and/or glint detection logic 102, according to variousembodiments. Processing logic 230 may be communicatively coupled toimage sensors 108 and 210, light sources 104, pairs of light sources206, display 224, and glint detection logic 102 to support operation ofHMD 204, according to an embodiment. Processing logic 230 may fully orpartially include glint detection logic 102. Processing logic 230 mayinclude circuitry, logic, instructions stored in a machine-readablestorage medium, ASIC circuitry, FPGA circuity, and/or one or moreprocessors.

FIG. 3 illustrates an example graph 300 of potential operationalcharacteristics that are based on intra-light source angle θ (shown inFIG. 1 ) between a light source 104A and a light source 104B, accordingto an embodiment. Graph 300 includes a signal-to-noise ratio (SNR) graphline 302 and an events rate graph line 304 plotted against changes inintra-light source angle θ. A type of light source that may be used inembodiments of the disclosure includes an LED, so intra-light sourceangle θ may be referred to as an intra-LED angle θ. As illustrated, whenintra-light source angle θ is approximately 2.5 degrees, pixel clustersof glints 306 become adjacent to each other while not overlapping. Asillustrated, pixel clusters of glints 308 are adjacent to each other butoverlap somewhat, which may result in poorer SNR and events rate, ascompared to that of pixel clusters of glints 306. One implementation ofhardware may achieve adjacent (while not overlapping) pixel clusters ofglints with at a 2.5 degree intra-light source angle θ. However, withdifferent hardware implementations another intra-light source angle θmay result in adjacent (while not overlapping) pixel clusters of glintsthat result in an increased SNR of captured glints.

FIG. 4 illustrates a process 400 for acquiring and using glintlocations, for example, in a head-mounted display or other ocularenvironment, according to embodiments of the disclosure. Process 400 maybe incorporated into glint detection system 100, ocular environment 200,eye tracking system 202, and/or HMD 204, according to embodiments of thedisclosure. The order in which some or all of the process blocks appearin process 400 should not be deemed limiting. Rather, one of ordinaryskill in the art having the benefit of the present disclosure willunderstand that some of the process blocks may be executed in a varietyof orders not illustrated, or even in parallel.

In process block 402, process 400 drives light sources 404 to emitdifferential light signals, according to an embodiment. The differentiallight signals can be defined by an illumination pattern that specifieswaveform shape, frequency of signaling, duty cycle, and wavelength oflight emitted. Process block 402 proceeds to process block 406,according to an embodiment.

In process block 406, process 400 receives, from image sensor 410, imagedata 408 that is representative of reflections of the differential lightsignals, according to an embodiment. The differential light signalsreflect off of the corneal surface of an eye and are referred to asglints. When the differential light signals change state (e.g.,high-to-low or low-to-high), the change in the image scene monitored byimage sensor 410 may register each changed portion of the image scene asan individual event. Image sensor 410 may be configured to register orcapture an event if an illumination change (positive or negative) in theimage scene changes by more than a pre-determined threshold value (e.g.,30-50 digital pixel values), according to an embodiment. Process block406 proceeds to process block 412, according to an embodiment.

In process block 412, process 400 identifies glint locations based onimage data 408, according to an embodiment. Process 400 may use machinelearning (e.g., neural networks) or may use 3D maps or models of glintvs. cornea orientation to identify an orientation of the corneal sphere,according to an embodiment. Process block 412 proceeds to process block414, according to an embodiment.

In process block 414, process 400 receives pupil data 416 using pupildetection hardware 418, according to an embodiment. Pupil detectionhardware 418 may include one or more light sources and/or one or moreimage sensors that are positioned and configured to capture an image ofa user's pupil, according to an embodiment. Pupil detection hardware 418may be configured to generate pupil data 416 using the differentiallight signals emitted by light sources 404, according to an embodiment.Process block 414 proceeds to process block 420, according to anembodiment.

In process block 420, process 400 identifies a pupil center based onpupil data 416, according to an embodiment. Process 400 may use pupildata 416 to identify a pupil center of an eye using one or more 3D mapsor models of an eye and/or using machine learning techniques (e.g.,neural networks), according to an embodiment. Process block 420 proceedsto process block 422, according to an embodiment.

In process block 422, process 400 provides glint location data and/orpupil center data 424 to eye tracking system 426, according to anembodiment. Process block 422 proceeds to process block 428, accordingto an embodiment.

In process block 428, process 400 determines corneal sphere orientationand/or gaze orientation based on glint location data and/or pupil centerdata 424, according to an embodiment. Process block 428 proceeds toprocess block 430, according to an embodiment.

In process block 430, process 400 provides corneal sphere orientationdata and/or gaze orientation data 432 to a display controller 434,according to an embodiment. Display controller 434 may be configured tocontrol images and/or user experience elements, focal points, etc. thatfor the display. The display may be a component of, for example, ahead-mounted display, according to an embodiment. Process block 430proceeds to process block 436, according to an embodiment.

At process block 436, process 400 adjusts display properties,application renderings, and/or user interface elements, based on cornealsphere orientation data and/or gaze orientation data 432, according toan embodiment. Process block 436 proceeds to process block 402, tocontinue to iterate through process 400, according to an embodiment.

FIG. 5 illustrates a process 500 for detecting glints, for example, inan eye tracking system, according to embodiments of the disclosure.Process 500 may be incorporated into glint detection system 100, ocularenvironment 200, eye tracking system 202, and/or head-mounted display204, according to embodiments of the disclosure. The order in which someor all of the process blocks appear in process 500 should not be deemedlimiting. Rather, one of ordinary skill in the art having the benefit ofthe present disclosure will understand that some of the process blocksmay be executed in a variety of orders not illustrated, or even inparallel.

In process block 502, process 500 drives a pair of light sources to emitdifferential light signals, according to an embodiment. Process block502 proceeds to process block 504, according to an embodiment.

In process block 504, process 500 receives image data from an imagesensor, wherein the image data is representative of reflections of thedifferential light signals off of the eye of the user, according to anembodiment. Process block 504 proceeds to process block 506, accordingto an embodiment.

In process block 506, process 500 determine an orientation of a cornealsphere of the eye at least partially based on the image data, accordingto an embodiment. Process block 506 proceeds to process block 502, torepeat process 500, according to an embodiment.

FIG. 6 illustrates a head-mounted device (HMD) 600, in accordance withaspects of the present disclosure. As described further below, inembodiments, HMD 600 may include a glint detection system that includeslight sources 104, image sensor 108, and glint detection logic 102, asdescribed above in connection with FIGS. 1-5 . An HMD, such as HMD 600,is one type of head mounted device, typically worn on the head of a userto provide artificial reality content to a user. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to the user, which may include, e.g., virtual reality (VR),augmented reality (AR), mixed reality (MR), hybrid reality, or somecombination and/or derivative thereof. The illustrated example of HMD600 is shown as including a viewing structure 640, a top securingstructure 641, a side securing structure 642, a rear securing structure643, and a front rigid body 644. In some examples, the HMD 600 isconfigured to be worn on a head of a user of the HMD 600, where the topsecuring structure 641, side securing structure 642, and/or rearsecuring structure 643 may include a fabric strap including elastic aswell as one or more rigid structures (e.g., plastic) for securing theHMD 600 to the head of the user. HMD 600 may also optionally include oneor more earpieces 620 for delivering audio to the ear(s) of the user ofthe HMD 600.

The illustrated example of HMD 600 also includes an interface membrane618 for contacting a face of the user of the HMD 600, where theinterface membrane 618 functions to block out at least some ambientlight from reaching the eyes of the user of the HMD 600.

Example HMD 600 may also include a chassis for supporting hardware ofthe viewing structure 640 of HMD 600 (chassis and hardware notexplicitly illustrated in FIG. 6 ). The hardware of viewing structure640 may include any of processing logic, wired and/or wireless datainterface for sending and receiving data, graphic processors, and one ormore memories for storing data and computer-executable instructions. Inone example, viewing structure 640 may be configured to receive wiredpower and/or may be configured to be powered by one or more batteries.In addition, viewing structure 640 may be configured to receive wiredand/or wireless data including video data.

Viewing structure 640 may include a display system having one or moreelectronic displays for directing light to the eye(s) of a user of HMD600. The display system may include one or more of an LCD, an organiclight emitting diode (OLED) display, or micro-LED display for emittinglight (e.g., content, images, video, etc.) to a user of HMD 600.

Embodiments of the invention may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,and any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto the viewer). Additionally, in some embodiments, artificial realitymay also be associated with applications, products, accessories,services, or some combination thereof, that are used to, e.g., createcontent in an artificial reality and/or are otherwise used in (e.g.,perform activities in) an artificial reality. The artificial realitysystem that provides the artificial reality content may be implementedon various platforms, including a head-mounted display (HMD) connectedto a host computer system, a standalone HMD, a mobile device orcomputing system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

The term “processing logic” (e.g., processing logic 230) in thisdisclosure may include one or more processors, microprocessors,multi-core processors, Application-specific integrated circuits (ASIC),and/or Field Programmable Gate Arrays (FPGAs) to execute operationsdisclosed herein. In some embodiments, memories (not illustrated) areintegrated into the processing logic to store instructions to executeoperations and/or store data. Processing logic may also include analogor digital circuitry to perform the operations in accordance withembodiments of the disclosure.

A “memory” or “memories” (e.g., memory 232) described in this disclosuremay include one or more volatile or non-volatile memory architectures.The “memory” or “memories” may be removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer-readable instructions, data structures, program modules, orother data. Example memory technologies may include RAM, ROM, EEPROM,flash memory, CD-ROM, digital versatile disks (DVD), high-definitionmultimedia/data storage disks, or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other non-transmission medium that can be usedto store information for access by a computing device.

A computing device may include a desktop computer, a laptop computer, atablet, a phablet, a smartphone, a feature phone, a server computer, orotherwise. A server computer may be located remotely in a data center orbe stored locally.

The processes explained above are described in terms of computersoftware and hardware. The techniques described may constitutemachine-executable instructions embodied within a tangible ornon-transitory machine (e.g., computer) readable storage medium, thatwhen executed by a machine will cause the machine to perform theoperations described. Additionally, the processes may be embodied withinhardware, such as an application specific integrated circuit (“ASIC”) orotherwise.

A tangible non-transitory machine-readable storage medium includes anymechanism that provides (i.e., stores) information in a form accessibleby a machine (e.g., a computer, network device, personal digitalassistant, manufacturing tool, any device with a set of one or moreprocessors, etc.). For example, a machine-readable storage mediumincludes recordable/non-recordable media (e.g., read only memory (ROM),random access memory (RAM), magnetic disk storage media, optical storagemedia, flash memory devices, etc.).

The above description of illustrated embodiments of the invention,including what is described in the Abstract, is not intended to beexhaustive or to limit the invention to the precise forms disclosed.While specific embodiments of, and examples for, the invention aredescribed herein for illustrative purposes, various modifications arepossible within the scope of the invention, as those skilled in therelevant art will recognize.

These modifications can be made to the invention in light of the abovedetailed description. The terms used in the following claims should notbe construed to limit the invention to the specific embodimentsdisclosed in the specification. Rather, the scope of the invention is tobe determined entirely by the following claims, which are to beconstrued in accordance with established doctrines of claiminterpretation.

What is claimed is:
 1. A glint detection system comprising: a pair oflight sources positioned on a head-mounted frame to illuminate a cornealsurface of an eye with differential light signals, wherein thedifferential light signals include two opposite light states thatconcurrently switch based on a pattern; an image sensor positioned onthe head-mounted frame to receive reflections of the differential lightsignals from the eye; and processing logic coupled to the pair of lightsources and to the image sensor, wherein the processing logic drives thepair of light sources to emit the differential light signals, whereinthe processing logic receives, from the image sensor, image data that isrepresentative of the reflections of the differential light signals,wherein the processing logic determines an orientation of a cornea ofthe eye at least partially based on the image data, wherein anintegration of a plurality of the reflections of the differential lightsignals produces a first cluster of pixels and a second cluster ofpixels, wherein a first of the pair of light sources is positioned adistance from a second of the pair of light sources so the first clusterof pixels is adjacent to the second cluster of pixels, wherein the firstcluster of pixels nearly touches the second cluster of pixels in animage map representation of the integration of the plurality of thereflections of the differential light signals.
 2. The glint detectionsystem of claim 1, wherein the image sensor is responsive to changes inbrightness in a field of view of the image sensor, wherein the imagedata is representative of the changes in brightness.
 3. The glintdetection system of claim 1, wherein the image sensor is an event camerathat is responsive to changes in brightness in a field of view of theimage sensor.
 4. The glint detection system of claim 1, wherein thedifferential light signals represent square-wave patterns of oppositepolarity.
 5. The glint detection system of claim 1, wherein the pair oflight sources emit infrared light.
 6. The glint detection system ofclaim 1, wherein the light sources emit the pattern at an illuminationfrequency of at least 1 kHz.
 7. The glint detection system of claim 1,wherein the pattern is a first pattern, wherein a first duty cycle ofthe first pattern emitted by one light source in a first of the pair oflight sources matches a second duty cycle of a second pattern emitted byone light source in a second of the pair of light sources.
 8. The glintdetection system of claim 1, wherein a first duty cycle of one lightsource in a first of the pair of light sources is less than 50% and asecond duty cycle of one light source in a second of the pair of lightsources is greater than 50% so that a sum of the first and second dutycycles is approximately 100%.
 9. The glint detection system of claim 1further comprising: one or more additional pairs of light sourcespositioned on the head-mounted frame to illuminate the corneal surfaceof the eye with one or more additional differential light signals. 10.The glint detection system of claim 1, wherein an intra-light sourceangle between the pair of light sources is at least 2.5 degrees.
 11. Theglint detection system of claim 1, wherein the pair of light sources arelight emitting diodes (LEDs), vertical external-cavity surface-emittinglasers (VCSELs), fiber optics, or out-coupling gratings.
 12. A headmounted display comprising: a display to provide display light; a lensassembly to transmit the display light from the display to an eyebox;and a glint detection system comprising: a pair of light sourcespositioned on a head-mounted frame to illuminate a corneal surface of aneye with differential light signals, wherein the differential lightsignals include two opposite light states that concurrently switch basedon a pattern; an image sensor positioned on the head-mounted frame toreceive reflections of the differential light signals from the eye; andprocessing logic coupled to the pair of light sources and to the imagesensor, wherein the processing logic drives the pair of light sources toemit the differential light signals, wherein the processing logicreceives, from the image sensor, image data that is representative ofthe reflections of the differential light signals, wherein theprocessing logic determines an orientation of a cornea of the eye atleast partially based on the image data, wherein a first duty cycle of afirst light source in the pair of light sources is less than 50% and asecond duty cycle of a second light source in the pair of light sourcesis greater than 50% so that a sum of the first and second duty cycles isapproximately 100%.
 13. The head mounted display of claim 12, whereinthe image sensor is responsive to changes in brightness in a field ofview of the image sensor, wherein the image data is representative ofthe changes in brightness.
 14. The head mounted display of claim 12,wherein the image sensor is an event camera.
 15. The head mounteddisplay of claim 12, wherein the differential light signals includesquare wave patterns that repeat at a frequency of at least 1 kHz.
 16. Amethod of corneal glint detection using differential lightingcomprising: driving, with processing logic, a pair of light sources toemit differential light signals towards an eye of a user to causediffuse surfaces on and around the eye to maintain a net illuminationthat is approximately constant, wherein the differential light signalsinclude two opposite light states that concurrently switch based on apattern, wherein the pair of light sources is positioned on a frame of ahead-mounted device to illuminate a corneal surface of the eye with thedifferential light signals, wherein a first duty cycle of a first lightsource in the pair of light sources is less than 50% and a second dutycycle of a second light source in the pair of light sources is greaterthan 50% so that a sum of the first and second duty cycles isapproximately 100%; receiving, with the processing logic, image datafrom an image sensor positioned on the frame of the head-mounted device,wherein the image data is representative of reflections of thedifferential light signals off of the eye of the user; and determining,with the processing logic, an orientation of a corneal sphere of the eyeat least partially based on the image data.
 17. The method of claim 16,wherein driving the pair of light sources includes driving the pair oflight sources to emit square waves having opposite polarities.
 18. Themethod of claim 16, wherein the image sensor is an event sensorconfigured to capture events from each pixel having a digital pixelvalue that exceeds a threshold.